feat: consolidated pip-compile -> uv migration (all 5 stacked PRs, rebased onto master) - #38915
feat: consolidated pip-compile -> uv migration (all 5 stacked PRs, rebased onto master)#38915irfanuddinahmad wants to merge 25 commits into
Conversation
|
Thanks for the pull request, @irfanuddinahmad! This repository is currently maintained by Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review. 🔘 Get product approvalIf you haven't already, check this list to see if your contribution needs to go through the product review process.
🔘 Provide contextTo help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:
🔘 Get a green buildIf one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green. DetailsWhere can I find more information?If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources: When can I expect my changes to be merged?Our goal is to get community contributions seen and reviewed as efficiently as possible. However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:
💡 As a result it may take up to several weeks or months to complete a review and merge your PR. |
|
Re: the Good catch, and worth noting: the org already ran a dedicated SHA-pinning sweep for this exact reason (openedx/.github#165, prompted by the tj-actions/changed-files supply-chain incident), but openedx-platform wasn't part of that ~121-repo effort. That said, I don't think pinning just |
…ration 1/5) Populates [project.dependencies] (from kernel.in + bundled.in), adds [project.optional-dependencies] for the legacy openstack storage backend, adds PEP 735 [dependency-groups] (coverage/testing/doc/assets/development/ semgrep/ci/dev, mirroring the current .in file composition), and [tool.edx_lint].uv_constraints + generated [tool.uv].constraint-dependencies for the ~20 repo-specific version pins, with a committed uv.lock. This is purely additive: the Makefile, tox.ini, and CI workflows are untouched and continue to use pip-compile/requirements/*.txt as the source of truth. Part of the pip-compile -> uv migration tracked in openedx/public-engineering#543 (1 of 5 PRs). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ions Found via real CI runs: a fresh uv resolution picked social-auth-core 5.0.2 (previously locked at 4.9.1 via pip-compile), which changes the OAuth pipeline's post-login redirect behavior and breaks common/djangoapps/third_party_auth's integration test suite (AzureAD, Google, LinkedIn, Twitter full-pipeline specs all failed the same assertion in tests/specs/base.py's assert_logged_in_cookie_redirect). This migration is meant to be a tooling swap, not a dependency upgrade, so pin back to the 4.x line rather than bundle an investigation into social-auth-core 5.x's behavior change into this PR. Mirrors the existing social-auth-app-django<=5.4.1 constraint, pinned for a related, already-deferred migration in this same dependency family. Follow-up tracked at #38841. Verified: all 46 previously-failing third_party_auth tests pass with social-auth-core==4.9.1 restored via this constraint. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Rewrites the Makefile's requirements targets, tox.ini, and ~13 CI workflows to use uv instead of pip-compile/pip-sync for the main app. Deletes requirements/edx/*.in and *.txt (superseded by pyproject.toml + uv.lock, added in PR 1 / #38835). requirements/constraints.txt, common_constraints.txt, and pip-tools.{in,txt} are intentionally kept for now: requirements/edx-sandbox and scripts/* still pip-compile against them and aren't migrated until PR 3/4. requirements/edx/{base,assets,development}.txt are regenerated as `uv export` compatibility artifacts (via the Makefile's compile-requirements target) since external tooling -- notably tutor's Dockerfile -- installs from those exact paths with plain pip, not uv. check_python_dependencies.yml is disabled (workflow_dispatch only, job gated with if: false) since find_python_dependencies can't scan pyproject.toml yet; tracked at openedx/repo-tools#725. User-confirmed before committing since this removes a CI safety net. Part of openedx/public-engineering#543 (2 of 5). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found via real CI runs after opening the PR (unit tests, Quality Others, and the ReadTheDocs build all failed): 1. Makefile's test-requirements used `uv sync --only-group testing`, which EXCLUDES [project.dependencies] entirely (confirmed: `--only-group` replaces the dependency set rather than adding to it, unlike `--group`). This meant Django, XBlock, and the rest of the actual application were never installed for test runs -- ModuleNotFoundError: No module named 'xblock' on every unit test shard. Fixed to `--no-default-groups --group testing`, matching what tox-uv itself generates for the equivalent tox environment. 2. .readthedocs.yaml had the same bug in its doc-requirements.txt export (`--only-group doc`). Since [project.dependencies] were excluded from that export, the subsequent `pip install -e .` resolved the whole dependency tree completely unconstrained by uv.lock's [tool.uv].constraint-dependencies -- picking setuptools==82.0.1 (violates setuptools<82) and Django==6.0.6 (violates Django<6.0). The setuptools violation broke fs/pyfilesystem2's pkg_resources import, crashing the Sphinx build via Django app loading. Fixed to `--group doc` plus `--no-deps` on the `pip install -e .` step, so dependencies only ever come from the properly-constrained export. 3. scripts/xsslint_config.py's SKIP_DIRS didn't exclude .venv. Under the old pip-compile system, dependencies installed into the system Python outside the repo checkout, so this never mattered. Now that `uv sync` creates .venv/ inside the checkout, xsslint's directory walk (which defaults to scanning the whole cwd) swept up thousands of vendored third-party files, inflating violations from 64 (the accepted baseline) to 316. Verified all three: real pytest run (59 passed) plus full Django `manage.py check` for both LMS and CMS against the corrected test-requirements; a local simulation of the RTD build job sequence confirms Django==5.2.15/setuptools==81.0.0 (both constraint-compliant) and a successful Sphinx build. Audited every other `--only-group` usage introduced in this migration (assets.txt compat export, semgrep.yml) -- both are intentionally project-dependency-free, matching the original files' documented behavior, and their CI checks already passed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
requirements/edx/{base,development}.txt were regenerated from the
uv.lock that predated PR 1's social-auth-core<5.0.0 constraint, so
they still referenced social-auth-core==5.0.2 -- caught by
check-consistent-dependencies.yml's re-run of `make compile-requirements`.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Gives requirements/edx-sandbox/ its own standalone pyproject.toml + uv.lock, independent of the main app's dependency graph (codejail intentionally runs untrusted code in a separate, isolated venv). [tool.edx_lint].uv_constraints holds only the subset of the root constraints relevant to this environment's deps (numpy, lxml, setuptools) -- uv/edx-lint have no cross-project constraint chaining equivalent to pip-compile's "-c ../constraints.txt", so root and sandbox constraints are now independently maintained (documented in requirements/edx-sandbox/README.rst). base.txt is regenerated as a `uv export` compatibility artifact (the README documents it as a supported, if unstable, direct pip-install target). releases/*.txt are untouched -- they're frozen historical snapshots, not part of any active compile loop; README now documents cutting future ones via `uv export` instead of pip-compile. Part of openedx/public-engineering#543 (3 of 5). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…on 4/5)
Gives scripts/xblock, scripts/user_retirement, and scripts/structures_pruning
each their own standalone pyproject.toml + uv.lock, mirroring the
codejail sandbox pattern from PR 3. structures_pruning's local
[tool.edx_lint].uv_constraints keeps the pymongo<4.4.1 pin it inherited
via the old "-c ../../../requirements/constraints.txt" chain.
These scripts are documented (in their own READMEs) to support git
sparse-checkout usage -- cloning only e.g. scripts/user_retirement/
without the rest of edx-platform. A self-contained pyproject.toml is
actually an improvement here over the old relative "-c
../../../requirements/constraints.txt" reference, which wouldn't even
resolve in a sparse checkout that excludes the root requirements/ dir.
Compatibility .txt exports are kept at their previously-documented
paths (e.g. scripts/user_retirement/requirements/{base,testing}.txt)
since each script's own README explicitly instructs `pip install -r`
against those exact paths.
Also fixes check-consistent-dependencies.yml's path filter and the two
PR-creating workflows' add-paths, neither of which would have picked
up changes to the new scripts/*/pyproject.toml or uv.lock files.
Part of openedx/public-engineering#543 (4 of 5).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Deletes requirements/constraints.txt, common_constraints.txt, and
pip-tools.{in,txt} -- these were kept alive through PR 2-4 because
requirements/edx-sandbox and scripts/* still pip-compiled against
them, but PR 4 was the last consumer, so they're now fully unused.
Removes the correspondingly-vestigial Makefile machinery: the
pre-requirements target, the COMMON_CONSTRAINTS_TXT curl-fetch-and-sed
target, and the CUSTOM_COMPILE_COMMAND/COMPILE_OPTS variables that only
existed to feed pip-compile invocations which no longer exist anywhere
in this repo.
Finalizes requirements/README.rst for the fully-migrated state and
fixes a couple of remaining stale references (constraints.txt ->
[tool.edx_lint].uv_constraints).
This is the last of 5 PRs migrating openedx-platform from pip-compile
to uv + PEP 621/735 pyproject.toml, tracked in
openedx/public-engineering#543. Two follow-up
items remain outside this repo's control:
- openedx/repo-tools#725: find_python_dependencies needs pyproject.toml
support before check_python_dependencies.yml can be re-enabled.
- Tutor's Dockerfile installs from requirements/edx/{base,assets,development}.txt
with plain pip; those are kept as `uv export` compatibility artifacts
(see PR 2 / #38836) rather than deleted, so no action is required there,
but tutor maintainers should be aware these are now generated files.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CI's "Compile requirements" check re-runs `make compile-requirements` and fails if it produces any diff, to catch exactly this kind of inconsistency. The compat-export files in this PR were originally generated with uv 0.11.26; a newer uv (0.11.30, matching what astral-sh/setup-uv installs in CI) resolves grpcio/grpcio-status with an explicit `; platform_python_implementation != 'PyPy'` marker that 0.11.26 omitted. Regenerated with uv 0.11.30 to match. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The uv migration for scripts/user_retirement dropped the lxml pin that requirements/constraints.txt previously carried, letting the lockfile resolve to lxml 6.1.1. The pin exists to avoid a libxml2 version mismatch at runtime (#36695), and this script transitively depends on lxml via simple-salesforce -> zeep, so the same pin applies here as it does at the repo root.
… group - Add a `bundled` dependency-group for the 15 packages (mostly third-party XBlocks) that master kept in a separate `bundled.in`, explicitly documented as "not core to the platform's functionality" -- this PR had flattened them into [project.dependencies] indistinguishable from actually-core deps. Include it from `testing` and explicitly sync/export it in base-requirements and the requirements/edx/base.txt compat export, so nothing that currently gets these packages (production images, test runs, CI) loses them. Verified via export/dry-run diffing: zero package or version changes to base-requirements, test-requirements, or the dev group as a result. - Remove `tox` from the `testing` group: it was duplicated with `ci` (which also has `tox-uv`, its required plugin), and nothing that syncs `testing` alone ever invokes tox directly. - Rename the `doc` dependency-group to `docs`, matching the `development` group's un-abbreviated naming and the docs/ folder; updated its one internal reference and .readthedocs.yaml.
- bundled -> non-core: clearer name for the "installed-by-default but
not required to run/test" group (matches the in-file comment wording).
- dev -> default: avoids near-duplicate naming with the development
group it wraps. Explicitly set tool.uv.default-groups = ["default"]
to preserve bare `uv sync`'s existing behavior of installing this
group, since renaming away from uv's built-in "dev" default name
would otherwise silently stop that.
- Verified both renames are lockfile/export no-ops: uv.lock package
versions are byte-identical (diff is metadata-only), and the
regenerated requirements/edx/{base,development}.txt compat exports
only changed in their header comments.
- Verified the assets group's minimal uv sync actually functionally
compiles Sass (not just installs packages): confirmed libsass's
_sass.compile_filename binding compiles real .scss to .css from that
minimal 4-package environment. The one compile failure encountered
(missing edx-bootstrap import) is from node_modules/npm, a separate
toolchain untouched by this dependency-group split -- documented in
the script's docstring.
…migration-consolidated # Conflicts: # requirements/constraints.txt # requirements/edx/base.txt # requirements/edx/development.txt # requirements/edx/doc.txt # requirements/edx/testing.txt
Merged master (bringing in automated Python dependency-upgrade PRs #38926/#38927), which required resolving modify/delete conflicts on the legacy pip-compile files this PR already deletes (constraints.txt, edx/doc.txt, edx/testing.txt -- kept deleted) and content conflicts on the edx/{base,development}.txt compat exports (kept ours, regenerated below). Bumped the edx-enterprise pin from 8.7.0 to 8.7.2 in [tool.edx_lint].uv_constraints to match master's automated bump, regenerated constraint-dependencies via edx_lint write_uv_constraints, re-locked (also picks up enterprise-integrated-channels 0.1.61 -> 0.1.65, matching master), and regenerated the base.txt/development.txt compat exports.
|
@irfanuddinahmad This commit changes are not related to PR so either they raised in conflicts resolution or AI tried to fix something by own. |
Revert the bundled -> non-core rename from a1d3ea1. The reviewer flagged (via Slack) that this broke the naming convention the rest of the migration follows: every dependency-group name traces back to its corresponding pre-migration requirements/edx/*.in file (testing.in -> testing, development.in -> development, doc.in -> doc, etc.), and requirements/edx/bundled.in is the actual source this group replaces. "non-core" has no such .in file to match. The dev -> default rename from the same commit is unaffected -- there was never a dev.in file, so that rename doesn't violate this convention and stays as-is. Verified lockfile/export no-op: uv.lock package versions are byte-identical (diff is metadata-only), and the regenerated requirements/edx/base.txt compat export only changed in its header comments.
…rence - static-assets-check.yml and docs/references/static-assets.rst were running/documenting `uv sync --group assets` (or bare `uv sync`), which doesn't disable `default-groups` and so installs the entire `default` superset (dev/test/docs/ci tooling) on top -- confirmed via CI logs this installed 399 packages for a job that only needs a handful. Scope both to `--no-default-groups` plus the specific groups actually needed (`assets` for building; `bundled` + `assets` for collectstatic, which needs the full app's installed Django apps). scripts/compile_sass.py's own docstring already documented and verified the minimal-install pattern this now matches. - requirements/edx-sandbox/README.rst referenced a `UV_SUBPROJECTS` Makefile variable that doesn't exist; the sub-project loop is inline in the `compile-requirements` target. Corrected the cross-reference.
|
|
||
| # The ^"? is because git may quote weird file paths | ||
| if git diff --name-only "$BASE_SHA" | grep -P '^"?((requirements/)|(scripts/.*?/requirements/))'; then | ||
| if git diff --name-only "$BASE_SHA" | grep -P '^"?((requirements/)|(scripts/.*?/requirements/)|(scripts/[^/]+/pyproject\.toml)|(scripts/[^/]+/uv\.lock)|(pyproject\.toml)|(uv\.lock))'; then |
There was a problem hiding this comment.
You should add a comment here, as this doesn’t seem to be a permanent solution. In the upcoming follow-up PRs, we plan to remove the requirements folder, so this will need to be updated again.
There was a problem hiding this comment.
Added a comment explaining this is transitional (requirements/edx-sandbox + the compat .txt exports still live under requirements/, but a planned follow-up per public-engineering#543 removes the folder entirely, at which point this pattern needs updating again). See 1c592ea.
|
|
||
| # The ^"? is because git may quote weird file paths | ||
| if git diff --name-only "$BASE_SHA" | grep -P '^"?((requirements/)|(scripts/.*?/requirements/))'; then | ||
| if git diff --name-only "$BASE_SHA" | grep -P '^"?((requirements/)|(scripts/.*?/requirements/)|(scripts/[^/]+/pyproject\.toml)|(scripts/[^/]+/uv\.lock)|(pyproject\.toml)|(uv\.lock))'; then |
There was a problem hiding this comment.
You should add a comment here, as this doesn’t seem to be a permanent solution. In the upcoming follow-up PRs, we plan to remove the requirements folder, so this will need to be updated again.
There was a problem hiding this comment.
Same fix applies here (duplicate thread) -- see 1c592ea.
| - name: Install Python dependencies | ||
| run: | | ||
| pip install -r requirements/edx/coverage.txt | ||
| pip install coverage diff-cover |
There was a problem hiding this comment.
why we are using pip here?
There was a problem hiding this comment.
pip should be replaced with uv
There was a problem hiding this comment.
Fixed -- this now uses uv sync --no-default-groups --only-group coverage --no-install-project --frozen, picking up the coverage dependency-group already defined in pyproject.toml for exactly this purpose. Restores the pinning that got lost when this was a bare pip install. See 1c592ea.
| - name: Install Python dependencies | ||
| run: | | ||
| pip install -r requirements/edx/coverage.txt | ||
| pip install coverage diff-cover |
There was a problem hiding this comment.
pip should be replaced with uv
…coverage tools, clearer no-op) - check-consistent-dependencies.yml: note that the requirements/ watch pattern is transitional -- requirements/edx-sandbox and the compat .txt exports still live there, but a planned follow-up (see public-engineering#543) removes the folder entirely. - unit-tests.yml coverage job: replace the unpinned `pip install coverage diff-cover` with `uv sync --only-group coverage`, using the `coverage` dependency-group that already exists in pyproject.toml for exactly this purpose -- restores the version-pinning/reproducibility the rest of this migration preserved everywhere else. - Makefile: clarify that `local-requirements` is an intentional no-op kept for backwards compatibility, not dead code, so an external caller relying on it doesn't read as silently broken.
|
Items not addressed in this PR:
|
- setup-uv installs and manages its own Python via the `python-version` input (sets UV_PYTHON, which uv resolves/downloads lazily on first use) -- actions/setup-python alongside it is redundant. Verified per file that no bare `python`/`pytest` invocation runs before the uv-managed venv is created and put on PATH (via the existing `echo "$PWD/.venv/bin" >> "$GITHUB_PATH"` step in each job), so this doesn't change what Python interpreter later steps actually get. js-tests.yml was missing that GITHUB_PATH line (harmless today since nothing in that job calls bare python, but added for consistency with every sibling job and defense against a future indirect call). - Bumped `astral-sh/setup-uv@v7` -> `@v9` everywhere it's used, for consistency (v9.0.0 is the current latest release). - Left the `coverage` job's `actions/setup-python` step in unit-tests.yml untouched -- not flagged, out of scope for this pass.
astral-sh/setup-uv stopped publishing floating major-version tags after v7 -- v8.x/v9.x only exist as full semver tags. @v9 broke every job in the previous commit ("Unable to resolve action astral-sh/setup-uv@v9, unable to find version v9"). Corrected to @v9.0.0 everywhere.
|
Quick correction to my last batch of replies: the v7→v9 bump broke every job ( |
Manual workflow testing neededTwo workflows changed in this PR are
|
…migration-consolidated # Conflicts: # requirements/common_constraints.txt # requirements/edx-sandbox/base.txt # requirements/edx/base.txt # requirements/edx/coverage.txt # requirements/edx/development.txt # requirements/edx/doc.txt # requirements/edx/semgrep.txt # requirements/edx/testing.txt # requirements/pip-tools.txt # scripts/user_retirement/requirements/base.txt # scripts/user_retirement/requirements/testing.txt # scripts/xblock/requirements.txt
| (cd $$d && uv export --frozen --no-hashes --group test --no-emit-project); \ | ||
| } > $$d/requirements/testing.txt; \ | ||
| done | ||
|
|
There was a problem hiding this comment.
The two loops over the same directory list can be merged into one — each export only depends on its own lock, not on the other directory. The second loop also has no error handling: a failed uv export silently writes an empty .txt and the target still exits 0.
Suggested replacement:
@# scripts/user_retirement and scripts/structures_pruning: base + testing (test group) compat exports.
@for d in scripts/user_retirement scripts/structures_pruning; do \
echo ; \
echo "== $$d ===============================" ; \
uv run --no-project --with edx-lint edx_lint write_uv_constraints $$d/pyproject.toml && \
(cd $$d && uv lock ${UV_LOCK_OPTS}) && \
{ \
echo "# GENERATED FILE, DO NOT EDIT DIRECTLY."; \
echo "# Compatibility export for anyone still 'pip install -r $$d/requirements/base.txt'"; \
echo "# directly instead of using uv. Source of truth: $$d/pyproject.toml / uv.lock."; \
(cd $$d && uv export --frozen --no-hashes --no-emit-project); \
} > $$d/requirements/base.txt && \
{ \
echo "# GENERATED FILE, DO NOT EDIT DIRECTLY."; \
echo "# Compatibility export for anyone still 'pip install -r $$d/requirements/testing.txt'"; \
echo "# directly instead of using uv. Source of truth: $$d/pyproject.toml (test group) / uv.lock."; \
(cd $$d && uv export --frozen --no-hashes --group test --no-emit-project); \
} > $$d/requirements/testing.txt \
|| exit 1; \
doneThree minimal changes from the current code: \ → && \ after uv lock, ; → && between the two export blocks, and || exit 1 after the last export.
| echo "# Source of truth: [dependency-groups].default in pyproject.toml / uv.lock."; \ | ||
| uv export --frozen --no-hashes --group default --no-emit-project; \ | ||
| } > requirements/edx/development.txt | ||
|
|
There was a problem hiding this comment.
The old REQ_FILES list included requirements/edx/testing, requirements/edx/doc, requirements/edx/coverage, and requirements/edx/semgrep — none of which have compat .txt exports here. If any CI configs, Dockerfiles, or external tooling still does pip install -r requirements/edx/testing.txt (etc.), they will get stale files silently. Either add the missing exports or confirm that all consumers have been updated to use uv sync directly.
| # | ||
| # GENERATED FILE, DO NOT EDIT DIRECTLY. | ||
| # Compatibility export for anyone still 'pip install -r scripts/user_retirement/requirements/testing.txt' | ||
| # directly instead of using uv. Source of truth: scripts/user_retirement/pyproject.toml (test group) / uv.lock. |
There was a problem hiding this comment.
This file (and scripts/structures_pruning/requirements/testing.txt) is kept as a compatibility export because units-test-scripts-user-retirement.yml and units-test-scripts-structures-pruning.yml still do a bare pip install -r .../requirements/testing.txt and were intentionally left unchanged in this PR.
What's the plan here? Migrating those two workflows to uv sync --no-default-groups --group test --frozen would let us delete these generated .txt files entirely. Is that tracked somewhere, or is it out of scope for this migration?
| requirements | ||
| scripts/**/pyproject.toml | ||
| scripts/**/uv.lock | ||
| scripts/**/requirements* |
There was a problem hiding this comment.
The add-paths block here is missing the root-level pyproject.toml and uv.lock — compare with compile-python-requirements.yml which correctly includes both. make upgrade-package edits [tool.edx_lint].uv_constraints in root pyproject.toml (via the tomlkit script above) and regenerates root uv.lock. Without these entries, the automated upgrade PR will be committed without the lock file changes, silently producing an incomplete upgrade.
| scripts/**/requirements* | |
| requirements | |
| scripts/**/pyproject.toml | |
| scripts/**/uv.lock | |
| scripts/**/requirements* | |
| pyproject.toml | |
| uv.lock |
There was a problem hiding this comment.
Confirmed and fixed -- compared directly against compile-python-requirements.yml's add-paths block, which does have both entries. Added here too. See 81670be.
| check_dependencies: | ||
| runs-on: ubuntu-latest | ||
| if: false | ||
|
|
There was a problem hiding this comment.
With both the PR/merge_group triggers removed and if: false on the job, there is now zero CI enforcement that contributors don't add unpinned git+https://... refs to pyproject.toml. The old system at least checked the requirements/edx/*.txt files on every PR. This is acknowledged as intentional pending openedx/repo-tools#725 / PR #735 — worth making sure that fix is tracked with enough urgency, since it's a supply-chain audit hole on every merged PR in the meantime.
There was a problem hiding this comment.
Fair concern, and agreed it's a real gap in the meantime. Checked the status of both: openedx/repo-tools#725 and the fix PR openedx/repo-tools#735 are both still open, not merged, so this isn't close to landing yet. I don't think there's a good in-repo mitigation for this PR specifically (re-implementing a pyproject.toml/uv.lock scanner here would duplicate what #735 is meant to provide upstream, and risks diverging from it) -- but I'll flag the urgency on the upstream issue directly rather than let it sit silently. In the meantime, requirements/README.rst's "Is there an unpinned git dependency?" section already documents the manual-review procedure as a stopgap.
| The following commands need to be run within a Python environment in | ||
| which requirements/edx/testing.txt has been installed. If you are using a | ||
| which the ``testing`` dependency group (``uv sync --only-group testing``) has | ||
| been installed. If you are using a |
There was a problem hiding this comment.
--only-group testing excludes [project.dependencies] entirely — this is the same bug caught and fixed in the Makefile's test-requirements target in an earlier commit (ModuleNotFoundError: No module named 'xblock'). This doc instruction will lead contributors into the same broken environment. Should match the Makefile's test-requirements target.
| been installed. If you are using a | |
| which the ``testing`` dependency group (``uv sync --no-default-groups --group testing``) has |
There was a problem hiding this comment.
Confirmed and fixed. Verified this precisely before applying your suggestion, since I wanted to know exactly what was missing rather than just take it on faith: uv sync --only-group testing --frozen --dry-run vs uv sync --no-default-groups --group testing --frozen --dry-run differ by 142 packages (205 vs 347). Diffing the two install sets shows the difference is real [project.dependencies] entries with no other reason to be pulled in transitively -- acid-xblock (explicitly annotated in pyproject.toml as "used for unit tests as well as manual tests"), analytics-python, bridgekeeper, codejail-includes, and others -- all absent under --only-group alone. Fixed to match the Makefile's test-requirements target exactly. See 81670be.
- upgrade-one-python-dependency.yml's add-paths was missing root pyproject.toml/uv.lock, unlike its sibling compile-python-requirements.yml. make upgrade-package edits both (via the tomlkit script and uv lock --upgrade-package), so without these entries the generated PR would silently omit the actual dependency-upgrade changes. - docs/concepts/testing/testing.rst told contributors to run `uv sync --only-group testing`, which excludes [project.dependencies] entirely (verified via `uv sync --dry-run`: acid-xblock, analytics-python, bridgekeeper, and codejail-includes -- all direct project dependencies, acid-xblock explicitly needed for tests -- are absent under --only-group but present under --no-default-groups --group, which is what the Makefile's test-requirements target actually uses). Fixed to match.
|
|
||
| - ``pip install -r requirements/edx/base.txt`` (production) | ||
| - ``pip install -r requirements/edx/development.txt`` (development) | ||
| - ``uv sync --no-default-groups`` (production) |
There was a problem hiding this comment.
Parity issue: base.in on master is -r kernel.in -r bundled.in, so base.txt has always included the bundled group. uv sync --no-default-groups only installs [project.dependencies] (kernel equivalent), silently dropping the bundled packages.
Suggestion
| - ``uv sync --no-default-groups`` (production) | |
| - ``uv sync --no-default-groups --group bundled`` (production) |
|
Could we add an explicit dependency mapping table to the PR description? The current description mentions files being "absorbed into pyproject.toml dependency-groups" but a reviewer has to trace through the code to figure out what replaced what. Something like this would make it self-contained:
The last row is especially worth calling out — it is the only intentional behavior change that affects individual developers and is currently invisible in the description. |
Summary
Migrates edx-platform from pip-compile/pip-tools to
pyproject.toml+uv, tracked in openedx/public-engineering#543.File-by-file summary
Root packaging & build config
[project.dependencies]list (previously just["setuptools"]), a new[dependency-groups]tree (bundled,testing,docs,assets,development,semgrep,ci,default),[tool.uv].default-groups/constraint-dependencies(machine-managed byedx_lint write_uv_constraints),[tool.edx_lint].uv_constraints(the hand-maintained version pins, each with a dated comment and issue link, carried over from the oldconstraints.txt), and a new[tool.coverage.*]tree.[tool.coverage.*]in pyproject.toml.pre-requirements/pip-synctargets replaced withuv sync --group ...targets;compile-requirementsrewritten to runuv lockfor the root project and loop over the 4 uv-managed sub-projects, then re-export compatibility.txtfiles at the old paths for external tools (e.g. Tutor's Dockerfile) that stillpip install -r requirements/edx/base.txtdirectly.runner = uv-venv-lock-runner+dependency_groups = testing; droppedqualityfromenvlist(quality now runs via a dedicated CI workflow/Makefile target, not tox) and the now-redundantusedevelop/commands_pre = make test-requirements.pip install -r requirements/edx/*.txttouv sync --group ....CI workflows
All follow the same mechanical pattern: drop the manual
pip cachesteps in favor ofastral-sh/setup-uv's built-in cache (enable-cache: true), replacepip install/pip check/pip freezewithuv pipequivalents, and addecho "$PWD/.venv/bin" >> "$GITHUB_PATH"so subsequent bare commands (pylint, tox, etc.) resolve inside the uv-managed venv.check-consistent-dependencies.yml: trigger-detection regex extended to also watchpyproject.toml/uv.lock/sub-project files, not justrequirements/.check_python_dependencies.yml: disabled (if: false), with a comment explainingfind_python_dependencies(edx-repo-tools) can't scanpyproject.toml/uv.lockyet; linked to the upstream tracking issue/fix PR.ci-static-analysis.yml,js-tests.yml,lint-imports.yml,migrations-check.yml,pylint-checks.yml,quality-checks.yml,semgrep.yml,unit-tests.yml: mechanical pip->uv swap as described above.compile-python-requirements.yml,upgrade-one-python-dependency.yml: updated to operate onpyproject.toml/uv.lockinstead of.in/.txt/constraints.txt. The dependency-downgrade script now edits[tool.edx_lint].uv_constraintsvia a propertomlkitTOML round-trip instead ofsed-patching a text file.static-assets-check.yml: same pip->uv swap, plus scoped the Python-deps-install step touv sync --no-default-groups --group bundled --group assets --frozen(rather than pulling in the fulldefaultgroup of dev/test/docs/ci tooling that this job doesn't need).units-test-scripts-structures-pruning.yml,units-test-scripts-user-retirement.yml: intentionally untouched -- they stillpip install -r scripts/.../requirements/testing.txt, and those compatibility files continue to be regenerated (viauv export) at the same paths, so these two workflows keep working unmodified.Documentation
.in/.txtworkflow.uv syncequivalents, scoped to the specific dependency-groups each workflow step actually needs.Dependency files (requirements/, uv.lock, scripts/*)
requirements/edx/*.in(all deleted): superseded by[project.dependencies]/[dependency-groups]in pyproject.toml.requirements/edx/base.txt,assets.txt,development.txt(kept, regenerated): machine-generateduv exportcompatibility exports at their historical paths for external tooling that still installs from them directly.base.txtexports[project.dependencies]plus thebundledgroup.requirements/edx/coverage.{in,txt},doc.{in,txt},testing.{in,txt},semgrep.{in,txt},bundled.in,github.in,kernel.in,openstack.txt,private.readme(all deleted): fully absorbed into pyproject.toml dependency-groups; no external tooling installed from these paths directly, so no compatibility export was needed.requirements/common_constraints.txt,requirements/constraints.txt(deleted): absorbed into[tool.uv].constraint-dependencies(machine-managed) and[tool.edx_lint].uv_constraints(hand-maintained) in pyproject.toml, with all original rationale comments and issue links preserved.requirements/pip-tools.{in,txt}(deleted): pip-tools itself is no longer needed.uv.lock,requirements/edx-sandbox/{pyproject.toml,uv.lock},scripts/{xblock,user_retirement,structures_pruning}/{pyproject.toml,uv.lock}(new): machine-generated lockfiles for the root project and its 3 independent standalone sub-projects (codejail sandbox, XBlock scripts, user-retirement scripts, structures-pruning scripts). Each sub-project sets[tool.uv] package = false(they're script bags, not installable packages) and carries only the subset of root constraints relevant to its own deps.scripts/*/requirements/{base,testing}.txt(kept, regenerated): same compatibility-export treatment asrequirements/edx/*.txt.Misc scripts
uv sync --no-default-groups --only-group assets --no-install-project), verified to actually compile Sass correctly with that reduced environment.pip install -r ...message updated touv sync --group default..venvto the linter's skip-dirs list.python.installpip-requirements mechanism with apost_create_environment/post_installhook that exports thedocsgroup viauv exportto a plainrequirements.txt, then lets RTD's own pip install it -- this preserves the constraint that Django/XBlock/etc. get resolved againstuv.lock's pins rather than picking up an unconstrainedsetuptools>=82that breaksfs/pyfilesystem2'spkg_resourcesimport.What's intentionally NOT done here (tracked externally)
find_python_dependenciesneedspyproject.tomlsupport beforecheck_python_dependencies.yml(disabled as part of this migration) can be re-enabled.requirements/edx/{base,assets,development}.txtwith plainpip. Those stay asuv exportcompatibility artifacts rather than being deleted, so no action is required on Tutor's side right now -- but Tutor maintainers should be aware these paths are now machine-generated, not hand-compiled.Verification
uv lockresolves cleanly (445 packages, root project); all 4 sub-projects (requirements/edx-sandbox,scripts/xblock,scripts/user_retirement,scripts/structures_pruning) sync cleanly withuv sync --frozen.make compile-requirementsend-to-end for the root project and all 4 uv sub-projects -- regenerated compatibility export files are byte-identical to what's committed, confirming consistency.🤖 Generated with Claude Code